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ABSTRACT 

Online education has grown dramatically over the past 15 years. At the university 
level, researchers have shown that online education has both its advantages - greater 
flexibility and access to student- and disadvantages - like disconnection with other 
students and faculty. 

Another possible drawback for the students enrolled in an online course is the 
inability to engage in reflective learning, or deep thinking. Some critics of online courses 
(Rahm and Reed, 1997 and Somer, 1999) suggest that online courses only offer passive 
learning and “one-way” information flows from teacher to student, significantly limiting 
a student’s ability to make connections within the material and critical apply the material 
to new situations. Alternatively, other researchers (Hay et al., 2005, for example) find no 
difference in reflective learning between an online versus on campus course, but also 
argue that students engagement in reflective thinking is much more instructor-dependent 
on line than on campus. 

This study extends the Hay et al. 2005 study by examining whether students who 
earned a good grade display more reflective thinking ability in an online course than 
mediocre or poor students. Students are asked at course-end the open-ended question 
“What advice would you give to a student just entering this course?” 

The results of this study suggest that reflective learning can take place within an 
online course - specifically a quantitative MBA course. But like the Hay, et al. 2005 
paper which labeled it dependent on the instructor, this study finds reflective thinking 
dependent on the student’s grade within the course. 
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INTRODUCTION 

Online education has exploded in growth since 1995. A 2010 Sloan Survey found 
that 5.6 million students were enrolled in at least one online course in the fall of 2009. 
The same survey showed that online enrollment had increased by 21 % from the prior 
year, while on-campus enrollment had only increased by 2%.This dramatic growth has 
been consistent throughout the past decade. Dolezalek (2003) found that expected 
increases in online courses would average 40% over the next decade. Earlier in the prior 
decade at many institutions, online education has become a replacement for on campus 
courses (Drago, Peltier and Sorenson, 2002). 

One of the major obstacles facing online education is the perception that the 
courses are not as rigorous as on campus course, or that the amount of material learned in 
the online course is not as much as the on-campus course. In the initial rush to create on 
line courses, some researchers found that courses lacked quality (Caudron, 2001), or that 
some colleges (mostly for-profit) created “diploma mills” (Hereford, 2000). On the other 
hand, other investigators have concluded that there were no significant differences in 
learning outcomes between online and on-campus courses (Palloff and Pratt, 2001) and 
that online students exhibited a higher level of engagement than on-campus students 
(Robinson and Hullinger, 2008). Employers have also signaled a belief in the 
comparability of online versus on campus education (Astani and Ready, 2010). 

One measure of quality of the educational experience is the students’ ability to 
reflect on the knowledge and experience within the course. This reflective ability is seen 
as a higher level of learning than mere memorization or application. It also has been 
shown to be particularly important to business education, where there is no defined 
solution (Schon, 1983, 1987) or uncertainty or change is imminent (Pee, et al., 2000). 

This paper examines whether students can demonstrate an ability to reflect on 
information given in three different quantitative online MBA courses at one university in 
the southwest United States - Introduction to Financial Accounting, Introduction to 
Managerial Accounting and Introduction to Statistics. An open-ended question was 
asked at the end of each course: “What advice would you give to a student entering this 
course?” Students were separated in those receiving a good grade (A/A-) versus a 
mediocre to poor grade (B+ or below). 

Using a Reflective Learning Dictionary in Wordstat, a text analysis program, the 
student advice was scrutinized for words of advice which would indicative of habitual 
learning, understanding and reflection. Students at all grade levels provide advice to 
students in similar amount and content in encouraging habitual learning and 
understanding. Good students, on the other hand, use more words to advice future 
students to also do reflect thinking, probably modeling their own learning experience in 
the course. 

This paper is outlined as follows. Section two provides a literature review 
describing some of the prior literature on reflective thinking in general and in online 
courses in specific. Section three provided the hypotheses and results. Section four 
discussed the results found in section three and extends the results further. Section five 
offers a conclusion. 
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LITERATURE REVIEW 

Reflection has been shown to be an integral part of education, and business 
education in particular. Past literature reviewed by Peltier, Hay and Diego (2005) and 
extended by Peltier, Hay and Diego (2006), defined reflection as . ..a series of 
increasing deep learning stages that may be used by individuals to integrate new 
knowledge, to contemplate its meaning and relevance to past knowledge, and culminating 
on whether to modifying existing beliefs and assumptions....” 

Reflection within business courses has also proved very desirable. Schon’s 
(1983, 1987) established the idea of a “reflective practitioner,” where a critical part of 
making an important business decision is the ability to step back, understand the “whys” 
and “wherefores” and integrated disparate pieces. Subsequently, others studies have 
shown reflective thinking to be important in areas such as business education (Reid and 
Parker, 1995) and accountancy (Velayutham and Perera, 1993). McFarlane (2001) points 
out the benefits of reflection in business ethics courses, and the idea of using learning 
logs as a retrospective benefit encourages students to look back in making connections to 
previously learned materials. 

The question of whether reflection is encouraged in an online business course is a 
valid one. Many researchers have concluded that online courses, completed in relative 
isolation of other students and the professor, have failed or at least limited the 
possibilities to question past learning or make connections between disparate parts of a 
course’s information. 

Studies have found that students are unhappy with the lack of connection with 
faculty (Arbaugh, 2002, and Blunt, 2001). Brower (2003) found that online courses 
might not facilitate higher level learning because a lack of rapid communication and 
feedback that can challenge prior knowledge or assumptions. For example, Shea and 
Boder (2001) pointed out that online courses can be seriously lacking in interactivity with 
students and professors and generally not as challenging as on campus courses. 

Problems include lack of feedback or timely feedback from instructors, too many 
students in online courses, projects poorly designed for an online environment - all 
deficiencies which could limit a student’s ability to apply newly learned material to a new 
situation. 

Higher level learning can be defined in a number of ways. For purposes of this paper, 
it is seen as through a continuum of learning, reaching a high enough state of knowledge 
to be able to reflection on the information learned in a new way - to be able to evaluate a 
unique situation, or to be able to critically think in new perspective. Hays et al. (2005) 
used this continuum in an examination of whether students enrolled in traditional (on- 
campus) courses demonstrate reflection/deep thinking more often or better than online 
students. From the students’ perspective, they find no difference “... we feel confident in 
being able to conclude that online management education is capable of encouraging 
higher levels of learning, thus countering critics who wrongly characterize online 
delivery as a passive delivery medium....” 
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The authors temper this conclusion with a caveat. While reflective learning can take 
place in both online and on-campus courses, the role of the lecturer/instructor is 
encouraging reflective thinking was much more critical in the online course. 

This study takes the examination of reflective learning in online courses further by 
examining the role of the student. Specifically in the advice given to future students by 
students completing the course, do students receiving higher course grades (A or A-) 
exhibit different thinking patterns (habitual vs. understanding vs. reflective) than students 
receiving lower grades (B+ or lower)? 

HYPOTHESIS AND RESULTS 

The continuum of reflective learning used in this study is as follows (as modified by 
Hays, et al., 2005). 

Habitual learning 

The lowest level is a non-reflective stage. Kember, et al., (2000) describe it as 
“.. .that which has been learnt before and through frequent use becomes an activity which 
is performed automatically or with little conscious thought....” 

At this level, specific tasks are treated as unrelated, memorization is emphasized, 
and little connection between activities is made. Students might encourage other students 
on how much to work to get a desired grade, what advice from the teacher to follow, or 
what information is important for grading or testing purposes. 

The key words a student might use to give advice encouraging habitual learning 
are found in Table 1. 

Table 1: Words which suggest advice for habitual learning 

• BASIC 

• EVALUATION 
•EXAM 
•FACT 

• FOLLOW 

• GRADE 

• INFORMATION 

• LITTLE 

• MATERIAL 

• MINIMUM 
•QUIZ 

• REMEMBER 
•TASK 
•TEST 

• MEMORIZATION 
•TIME 

• WITHOUT_THINKING 
•WORK 

• WORKLOAD 
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Understanding 

The second level is also a non-reflective state. Again, Kember, et al. label this 
stage as ^‘...understanding focuses on comprehension, without relation to one’s personal 
experiences, in that the learner need only focus on comprehending the material...” 

Students are this level might encourage future students on how to comprehend the 
material either through doing homework, or through repetition of problems, or learned or 
learning process. They also might tell students to apply the material learned or to 
comprehend it in a certain way. 

The key words a student might use to give advice encouraging understanding are 
found in Table 2. 

Table 2: Words which suggest advice for understanding 

• APPLICATION 

• APPLIED 

• APPLY 

• COMPREHEND 

• COMPREHENSION 
F- CONCEPT 

• HOMEWORK 

• LEARNED 

• REPETITION 

• REPETITIOUS 

• THEORY 

• UNDERSTAND 

• THINK 

Reflection and Critical Reflection 

The third and fourth level is a reflective state of thinking and doing. Luang and 
Kember (2004) show that there is a causal relationship between deep thinking and 
reflecting - the former causing the latter. It is characterized by challenging assumptions, 
face new scenarios and using real-world experiences to make connections. 

A current student advising future students in the course to engage in reflective 
learning might suggest challenging alternatives or assumptions or beliefs, or search for 
deeper meanings or new or novel ideas. He or she might also tell prospective students to 
explore the implications of actions, or to discover or be aware or reflection on new 
information. 

The key words a student might use to give advice encouraging reflection are 
found in Table 3. 
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Table 3: Words which suggest advice for reflection 

• ALTERNATIVE 

• ASSUMPTION 

• AWARE 

• BELIEF 

• BELIEVE 

• CHALLENGE 

• CHANGE 

• DECISION 

• DILEMMA 

• DISCOVER 

• EXPERIENCE 

• ALTERNATIVE 

• FUTURE 

• EXPLORE 

• IMPLICATION 

• IMPROVE 

• IMPROVEMENT 

• INTEGRATE 

• INTERACTION 

• MEANING 

• NEW_IDEAS 

• NOVEL 

• PERSPECTIVE 

• PREMISE 

• QUESTION 

• REAPPRAISE 

• REASSESS 

• REFLECT 

• REFLECTION 

• RETHINK 

• SELF-AWARE 

• TRANSFORM 

During a three year period (2008-2011) in a large MBA program at a private 
university in the US, the following question was asked at the end of each online course in 
Introduction to Financial Accounting, Introduction to Managerial Accounting and 
Introduction to Statistics: “What advice would you give to students entering the course?” 
Bonus points on the final exam were offered for an “attempted” response (which 
averaged around 30 words). Responses were received from 313 students over nine 
sections with six different instructors. 

Table 4 shows the distribution of gender by course grade level. A chi-square 
analysis indicated an insignificant relationship between the two variables of good grade 
versus mediocre/poor grade ( x 2 — ##, df — 3,p > .05). 


Does reflective learning, Page 6 



Journal of Instructional Pedagogies 


Table 4: Distribution of respondents by gender/grade 


Good Grade 
Mediocre/Poor Grade 


Male 

145 

69 


Female 

59 

40 


In previous research on the assessment of reflection, a quantitative approach was 
taken. Typically students completed a questionnaire or were evaluated on a behavioral 
checklist (Boenink, Oderwald,De Jonge, Van Tilbugh,&Smal, 2004; Kember, Leung, 
Jones, Loke, McKay, Sinclair, &Tse, 2000; Sobral, 2000; Van Woerkom, Nijhof, 
&Nieuwenhuis, 2002). In this research, a qualitative analysis approach was used. This 
approach has been used to assess Student Evaluation of Teaching (Rovai, Ponton, 

Derrick, & Davis, 2006; Kelly, Ponton, &Rovai, 2007). Responses to the open-ended 
questions were analyzed for word use indicative of the three levels of learning: habitual, 
understanding, and reflective. 

A Learning Level Dictionary was developed using the key sets of words noted in 
Tables 1, 2 and 3 to show student advice within habitual learning, understanding and 
reflection and critical reflection. 

Next, the additional literature on the assessment of the three learning levels was 
scanned to identify words that were descriptive of each learning level (Boenink, et al, 
2004; Kember, et al., 2000; Sobral, 2000; Van Woerkom, 2000). Key words were 
extracted and allocated to one of the learning levels. For example, habitual learning is 
often described as memorization-oriented. Therefore, variants of the word memorize 
were allocated to the dictionary’s Habitual Learning section. 

In the analysis phase, responses to the open end question were compared to the 
words in the three categories of the Learning Level Dictionary. The rationale behind the 
qualitative analysis is that an individual’s thought level can be shown in the words they 
used in their open-ended responses. 

If students are able to use all levels of learning, including reflective learning/deep 
thinking, they should be providing advice to future students using the words in the three 
categories of the Learning Levels Dictionary. On the other hand, students receiving lower 
grades may not have been able to reach the higher levels of thinking, specifically using 
reflections or critical reflections. If this is the case, then the key words used by students 
receiving good grades will be different than those receiving mediocre or poor grades. Of 
course, this could also be the case over all three areas - good students will also give 
advice using more words that encourage habitual learning, understanding and 
reflections/critical reflections. 

This leads to three hypotheses: 

HI: Students who receive a good grade (A/A-) in the course will encourage students 

to do more habitual learning than student who received a mediocre or poor grade (B+ 

or less) in the course. 

Using the chi-square test, the following results are noted in Table 5: 
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Table 5: Student course performance and habitual learning advice 

G M/P Chi2 P-value 
Habitual Learning 265 112 4.348 0.114* 

* not statistically significant 

Hence HI cannot be rejected. Therefore, there is no significant difference in course 
advice given when students in the current course give advice to future students on 
habitual learning. Good students and mediocre to poor students each advice future 
students similarly in areas of habitual learning. 

H2:Students who receive a good grade (A/A-l in the course will encourage students 

to do more understanding of material in the course than students who receive a 

mediocre or poor grade (B+ or less) in the course. 

Using the chi-square test, the following results are noted in Table 6: 

Table 6: Student course performance and understanding advice 

G M/P Chi2 P-value 
Understanding 157 74 .7972 0.673* 

* not statistically significant 

Like HI, H2 cannot be rejected - students with poor or mediocre grades use the same 
types of advice that encourages understanding as good students. 

H3: Students who receive a good grade (A/A-) in the course will encourage students 

to do more reflections on relationships in the course than students who receive a 

mediocre or poor grade (B+ or less) in the course. 

Using the chi-square test, the following results are noted in Table 7: 

Table 7: Student course performance and reflective learning advice 

G M/P Chi2 P-value 
Reflection 217 71 13.218 0.001* 

* very statistically significant 

H3 can be accepted - good students encourage future students to use reflection in their 
advice more so than mediocre or poor students. 
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While it can be shown that students within on-line quantitative courses can be 
reflective thinkers, the next question is “why only good students?” In other words, what 
can faculty within courses do to encourage all students, not just those that are earning 
higher grades, to engage in higher level reflective thinking. 

One possible way to restrict quantitative online courses to more “online 
experienced” students - the plausible belief being that students who already 
understanding the expectations and drawbacks of an online course (working alone, 
limited student/faculty interaction, self-motivation requirements) will be more able to 
focus on learning the material in the course particularly well. 

That leads to the fourth hypothesis: 

H4: Students who have completed 2 or more online courses at the MBA level before 

this course will encourage students to do more reflective learning than students who 

have only completed one or no online courses before this course. 

Using the chi-square test, the following results are noted in Table 8: 

Table 8: Prior online course experience and encouraging reflective learning 

2+ 0-1 Chi2 P-value 
Reflection 13 21 .0000 1.000* 

* not statistically significant 

Hence H4 cannot be proven - students with more experience taking online courses in 
the MBA program before this course do not necessarily encourage more reflective 
thinking to future students. 

A second possibility is that students who are accounting or finance concentrations 
will engage in higher level reflective thinking since they will be mindful of the relevance 
of the material in the course to their future career aspirations and be looking for way to 
integrate that knowledge. 

That leads to fifth hypothesis: 

H5: Students who have declared concentrations in accounting or finance at the MBA 

level before this course will encourage students to do more reflective learning than 

students who have are concentrating their MBA studies in some other area. 

Using the chi-square test, the following results are noted in Table 9: 

Table 9: Declared MBA concentration and encouraging reflective learning 

Hi Lo Chi2 P-value 
Reflection 6 28 .0051 0.975* 

* not statistically significant 
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Thus H5 cannot be proven - the advice given by student who are MBA 
concentrations in accounting or finance are no different than students in any other MBA 
concentration area. 

CONCLUSION 

This study examines the wording in advice given by students completing three 
different MBA online courses by asking the open-ended question “What advice would 
you give to someone entering this course?” 

Using a Reflective Learning Dictionary in Wordstat, a text analysis program, 
words indicative of habitual learning, understanding and reflection were each noted and 
summarized. Students receiving a final grade of A or A- encouraged students to engage 
in reflective learning more than students whose final grade was lowered than A-. No 
significant differences were noted between good versus mediocre/poor students in advice 
which would encourage habitual learning or understanding. 

The study was extended to see if students would encourage more reflective 
learning if they had significant prior online course experience within the MBA program, 
or if their concentration was in a quantitative area, such as accounting or finance. No 
significant differences were noted. 

Future studies in the area should examine why some students perform better than 
others - it is merely aptitude or motivation or organizational skills? Do any of these 
allow or encourage students to be able to have the deep learning necessary for reflection? 
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